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Oral evidence 


Education and Skills Committee: Evidence Ev 1 


Taken before the Education and Skills Committee 


on Monday 10 July 2006 


Members present 


Mr Barry Sheerman, in the Chair 


Mr Douglas Carswell 
Mr David Chaytor 
Jeff Ennis 

Paul Holmes 


Helen Jones 

Mr Gordon Marsden 
Stephen Williams 
Mr Rob Wilson 


Witnesses: Sir Alan Wilson, Director-General for Higher Education, DfES, and Professor David Eastwood, 
Vice-Chancellor, University of East Anglia and Chief Executive designate of the Higher Education Funding 


Council for England, gave evidence. 


Q1 Chairman: I think you are both aware of the 
great interest there has been since the Chancellor of 
the Exchequer made a reference to the winding up of 
the RAE exercise and its replacement. There was 
some surprise that the Chancellor had made that 
announcement rather than anybody else, but we will 
come to that later. Looking at this Committee’s 
work on the funding of higher education and who 
should pay for it two years ago, it did play some part 
in the resolution of our discussions over variable fees 
and all that, which many people think was a very 
important milestone in the development of higher 
education in our country. Equally, if we do not get 
the research side and the funding of research right in 
this country, again that has serious implications for 
our university system. All this seems to be a bit of a 
rush. Somebody said to me at a recent conference 
that I chaired at the Royal Society—we seem to have 
most of the Vice Chancellors of the country at it— 
that this all stems from prior to the Chancellor’s 
statement that three leading vice-chancellors went to 
see either the Chancellor or other people in the 
Treasury and seemed to persuade them that we 
needed some changes fast. Sir Alan, I cannot believe 
that that was the case, but you know how these 
stories emerge. Sir Alan, why is there haste about all 
this? It seems all to be in a bit of a hurry. 

Sir Alan Wilson: In terms of stories about three vice- 
chancellors, that is something I know nothing of, if 
it ever took place, so I start from that position. 


Q2 Chairman: So the Vice-Chancellors of Imperial, 
University College London, and Bristol did not go 
and see the Chancellor! 

Sir Alan Wilson: Well, nothing is impossible, 
Chairman, but I have no knowledge of it. That is all 
I am saying. In terms of haste, the notion that it was 
all very fast for that kind of reason we would say was 
not the case, partly because the document that was 
eventually produced is a follow-up to a document 
that was published in 2004, the original 10-year 
framework for investment in science. The policies 
that were further developed in the budget science 
paper were really a continuation of the policies that 
were announced in July 2004. From our perspective 


it has been continuing work. In terms of the Next 
Steps paper that was published with the budget— 
and in a sense this almost answers the question, 
“why the Chancellor?”—1t goes back to the 10-year 
science framework. I think the Chancellor is 
anxious, as part of the budget, as I understand it, to 
have a comprehensive review of progress since the 
2004 paper, and research was part of that. From our 
point of view it is an ongoing process, and we have 
worked with HEFCE all the way through that 
period in terms of looking at possible metrics and 
performance indicators. We have talked to Treasury 
officials and DTI officials. Much of what was 
presented in the press about the rush, and certainly 
the story about when DfES officials told them are 
simply not true. 


Q3 Chairman: So it is not true that you were 
surprised in the DfES! Professor Eastwood, would 
you know whether this was greeted with surprise in 
HEFCE? 

Professor Eastwood: | think there is a parallel story 
to the one that Sir Alan has just sketched. After the 
RAE 2001 the funding councils jointly set up a 
review of the RAE methodology under Sir Gareth 
Roberts; and on the basis of the Roberts 
recommendations, the funding councils agreed 
substantial changes to the methodology for 2008, 
including a substantial reliance on metrics in the 
2008 exercise. At nearly the same time the decision 
was taken alongside the RAE in 2008 to run a 
shadow metrics exercise; that is to say to test in real 
time an alternative lighter touch methodology for 
research assessment. Indeed, work was in hand 
within the funding council, and between the funding 
council and other bodies, to build that alternative 
model. So there was a direction of travel here 
towards a robust RAE in 2008, on the basis of what 
we might broadly call the Roberts methodology; but 
alongside that to test and chart a new future for 
research assessment in the world beyond 2008. To 
that extent, what was announced at the budget and 
the announcements around the budget were 
consistent with that direction of travel. 
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Q4 Chairman: Professor Eastwood, was it not the 
case that at the time of the Gareth Roberts report 
something like 80% of institutions expressed 
approval and satisfaction with the peer review and 
the RAE exercise generally. Was that not the case at 
the time? 

Professor Eastwood: \t is certainly the case that in 
response to the consultation around Roberts there 
was a strong preference within the sector to retain a 
significant element of peer review, and that Roberts 
did; but alongside that there was the move towards 
a greater reliance on metrics, and a sense too that the 
available metrics would continue to develop both in 
terms of range and in terms of reliability as time 
moved on. I do not think that the position we are 
now in is anything more than an evolution—a 
substantial evolution perhaps but nevertheless an 
evolution of the position we were in around 
2002-2003. That is reflected in the kinds of responses 
that are beginning to emerge from the sector. Of 
course, there was a flurry of excitement when the 
consultation document was published, and one 
would expect that; but I think there was a serious 
engagement with the issues raised within the 
consultation document, and something close to a 
settled view in the sector that the RAE 2008 is very 
important—it is very important that we get it right 
for a whole series of reasons—but that this would be 
the last RAE “in the current form”. 


Q5 Chairman: To be honest about this, it is all about 
who gets the money, is it not? Whatever system you 
use, it is about who gets the money to conduct the 
research. Is it not the fact that the sensitivity is that if 
you change the rules you may be taking money away 
from one set of institutions or departments and 
giving them to others? At the heart of this is there 
someone in HM Treasury or someone in the higher 
education world or someone in the Department for 
Education and Skills saying, “the money is going to 
the wrong people”? Are they saying that? Have we 
got to change it? 

Professor Eastwood: From the perspective of the 
Funding Council, a research assessment does three 
things. First, it identifies and assesses research 
quality, which is central to the Funding Council’s 
commitment to fund excellence where it finds it. On 
the basis of that, and importantly, it has constituted 
a very important benchmark for the quality of 
research within UK HE. That matters not just in 
terms of research and research performance 
internationally, but also in relation to the branding 
of UK higher education. Thirdly, as you say, 
Chairman, it is an exercise which underpins the 
funding allocations that the Funding Council makes 
to institutions. We could have a long discussion 
about what might constitute the right kind of 
distribution funding, but I have heard nothing in the 
current debate that suggests that the broad 
allocation of funding is inappropriate. It needs to be 
dynamic and it will shift over time. There are a 
number of rather important debates around the 
funding of interdisciplinary and multidisciplinary 
research for example, and also about the resourcing 
of applied research. There are some areas where 


some concerns have been expressed, and expressed 
with some force. It is a large step from that to saying 
that there is profound dissatisfaction at the funding 
allocations that are emerging; on the contrary, the 
Next Step document was rather presuming that the 
funding allocations would remain broadly as they 
were. 


Q6 Chairman: Sir Alan, is it not the fact that if there 
was a discontent about the way research funding was 
being allocated—we have seen a dramatic change 
over the years in the number of 5-star and 5 
departments; the number has increased very rapidly, 
and that may because research is so much better or 
because academics have learnt to play the game 
rather better—or more academics have learnt to 
play the game rather better. What is at the heart of 
this haste? Everybody knew that there was a change 
in the system going on. Everybody knew that in 
Australia they were moving from a metrics system 
and planning to move to a peer-reviewed RAE 
system—Hong Kong similarly. There seems to be a 
movement in the other direction. We seem to be 
going into this metrics area against the flow—is 
that right? 

Sir Alan Wilson: I think it is against the flow in the 
sense that we have had twenty years’ experience of 
the research assessment exercise. 


Q7 Chairman: Which everybody thinks is 
wonderful, and they are copying us! 

Sir Alan Wilson: They think that what has been 
achieved in this country since 1986 is very 
impressive. It may be that they have to proceed from 
the equivalent of a time base’ that we may have 
achieved 10 years ago or something of that kind. Not 
uncontroversially—and in that sense you must be 
right—there has been a reasonable assumption that 
a new method of both assessment and allocated 
funds would be appropriate. In terms of your 
original question, there has not been a position in the 
DfES or as far as I know anywhere else saying that 
certain kinds of universities should get more money 
and some should get less, because at the government 
level it has always been about policy. Professor 
Eastwood has indicated some of the current issues 
about funding in applied research, funding of 
interdisciplinary research; but the principle that the 
best research will be funded wherever it is, is 
something that underlines all of this. The scale of the 
exercise—which is why we look for a system that is 
less bureaucratic—is considerable. It is not just 
simply that the cost is measured—which is 
substantial but reasonable in relation to what has 
been allocated—it is the time and, in a sense, the way 
that it dominates the development of policies in 
particular institutions. If there is a simpler way of 
doing it, there is the possibility, as we said in the 
consultation paper, that we might be able to move 
from what is a fantastic platform that has been 
established to being ever more ambitious in the 
future. 
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Q8 Chairman: So you do not think there is any truth 
in the assertion that the Chancellor and HM 
Treasury might have been saying, “we want more 
applied research, more technology transfer, more 
team working across departments and across 
universities; we want to see research much more 
shaped towards what increases the wealth of the 
United Kingdom and much more practical 
outcomes”. Do you think that has not been a 
Treasury view? 

Sir Alan Wilson: \t may be a Treasury view in the 
sense that those concerns have been shared right 
across the sector and right across different funding 
agencies. Even the research councils, which you 
might say are primarily there to support basic 
research, have had an increasing concern with the 
mechanisms with which the results of that basic 
research are applied. I think that many would argue 
that the distance in time in terms of what used to be 
called a linear model between basic research and 
becoming useful is shrinking. I think it is a policy 
question not just for HM Treasury but for all of us— 
how come the funding needs of research in the 
economy as well as blue-skies research and research 
in all kinds of public interests are balanced. At the 
end of the day it may go back to funding in another 
sense; that generous though the research budget is in 
this country relatively, there is never enough to 
sustain what everybody would like to sustain. 


Q9 Mr Wilson: This is not about a new way of 
handling research; this is about the Chancellor 
saving money, is it not? This is about HM Treasury 
making a grab for £45 million in savings. 

Sir Alan Wilson: | think that all the evidence in terms 
of the Chancellor’s support of science is that he has 
been committed to increasing budgets rather than 
saving money on research. In fact in the last two 
spending reviews he has actually ring-fenced greater 
than average increases for research funding, and 
certainly there is no evidence in any discussions I 
have had with officials that that situation is 
changing. 


Q10 Mr Wilson: In 2008 we are all going to see 
tighter times ahead in education. HM Treasury 
needs to save money. They are looking forward to 
where they are going to save it, and this is just one of 
a number of areas that they are targeting, is it not? 
Sir Alan Wilson: 1 do not want to anticipate 
discussions on the comprehensive spending review 
that will take place in all departments, and certainly 
within the DfES; and it is a matter of political 
judgment for our ministers at the end of the day to 
decide on these relative priorities. I would say again, 
Chairman, that I have no evidence that in any of the 
government departments that are party to Next 
Steps, DfES, HM Treasury, Health and DTI, that 
anybody wants to do anything but sustain a strong 
research base and provide the funding for it. 


Q11 Mr Wilson: As the Chairman earlier indicated, 
there are high levels of satisfaction with the current 
system, so why does it need to be replaced? 


Sir Alan Wilson: In a sense, as I think I indicated 
earlier, the big savings for the community are less in 
terms of money, wherever the money savings are 
channelled—and they could be channelled into 
further research—it is the particular way in which it 
has dominated the time of many academics, and 
there is a good possibility, I would judge—and I am 
perhaps making a personal comment, Chairman— 
that it could increase research productivity. 
Professor Eastwood: The direct costs of RAE 2008 
will be of the order of £8 million, the direct costs to 
the Funding Council. The £45 million is a 
calculation of indirect costs incurred in institutions 
in preparing for the RAE. Some of those costs are 
constant costs, costs associated with research 
management and performance management and so 
forth. I think there are almost certainly savings to be 
made here. The system has, as Sir Alan said, 
matured over twenty years, and some of those 
embedded costs can probably be stripped out. I 
would echo what Sir Alan said: that would be a 
saving that I see being redirected into the research 
effort, rather a saving that was stripped out of the 
HE budget. 


Q12 Mr Wilson: The consultation paper assumes 
that the RAE should be discontinued, and it seems 
to be on the basis of widely held views, or what 
people say. Where is the substance? You yourself 
said it is time taken by academics. Where is the 
substance? Where is the evidence for the supposition 
that you are making? 

Sir Alan Wilson: There is evidence that it is possible 
in principle to run a simpler system because of the 
correlations between the data reported in the 
consultation paper. 


Q13 Mr Wilson: Where do I actually see that 
evidence? 

Sir Alan Wilson: There are the models that are on the 
website, where the reference was given in the 
consultation paper. Indeed, in the Next Steps paper 
there are two graphs on pages 20 and 21 of chapter 
4, which show in aggregate levels certainly some very 
high levels of correlation. In terms of the feasibility 
of using something like research income and then 
other indicators as measures of quality that can then 
be used for funding allocations, the evidence is there, 
and quite a number of people have believed for a 
long time that this would produce a simpler method 
that would free people for research, rather than run 
the process as it has been run. As we are all agreeing, 
none of this is without controversy. Any proposals 
will be controversial, and in any consultation there 
will be people who say, “Keep the RAE; it has 
worked very well”; and there will be others who will 
say, equally strongly, “the RAE is now in a 
diminishing returns phase; there is a simpler way of 
doing it; please let us do that”. 
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Q14 Mr Wilson: You are right that it is extremely 
controversial, and for that reason do you not think 
the Government should have made the case for 
change a lot more strongly than it has? 

Professor Eastwood: Some of the case was made in 
Next Steps chapter 4. There is an analysis there, and 
the consultation document rather presumed on that 
and did not wish to replicate it. The other thing I 
would say about the document that was published at 
the beginning of June is that it is genuinely a 
consultation document. It offers some illustrations 
in terms of types of models and in terms of funding 
outcomes of those models. It is a consultation with 
the sector, asking the sector to engage with the issues 
and with the analysis. I think that those involved in 
this process are fully expecting that the sector will 
come back, obviously with the critical engagement 
of the kind that you are suggesting; but also will 
come back with proposals that will take us forward 
in maintaining the capacity to assess research 
quality, to have a_ sensible framework for 
distribution of research funding, but to do so in a 
lighter-touch way than has been possible in the last 
few years. 


Q15 Mr Wilson: But it is not a consultation that 
keeps the RAE system on the table; it is a 
consultation on a metrics basis, is it not? Therefore 
it is not a consultation at all. 

Professor Eastwood: It is a consultation that asks a 
certain set of questions around the STEM subjects 
on the one hand, and the arts and humanities and 
cognate disciplines on the other; so it does recognise 
that methodologies for different disciplines may 
vary. To that extent it recognises that there is a 
varied disciplinary landscape with which research 
assessment has to engage. It is also asking whether 
or not we have the right sorts of metrics, or whether 
the sorts of methodologies we are envisaging will 
enable us to make appropriate assessments. You are 
right that it is not saying, “Do you wish to maintain 
the RAE in its current form?” However, it is I think 
inviting serious engagement, and it is also inviting 
respondents to think about proposals which might 
further elaborate the sorts of methodologies that are 
implicit in that document. A number of institutions, 
not least the research-intensive institutions, are 
beginning to engage with the consultation in 
precisely that spirit. 


Q16 Mr Wilson: What are you actually trying to 
achieve? What is the end game for this metrics-based 
system? What is it that you want it to do that the 
system is not doing already? 

Sir Alan Wilson: It will be less bureaucratic; it will 
cost less; it will take less staff time, but still support 
excellence; and the correlations that I have talked 
about suggest that it can still support excellence. It 
encourages ambition, or facilitates ambition, by 
taking away people from the formalities of the RAE. 
It responds to what is becoming an increasingly 
important interdisciplinary research agenda. It 
connects to the subject that Professor Eastwood 
talked about, which is the relative amount of 
support for applied research or user-led research. 


That is very important for all of us. The extent to 
which metrics relate to that should, at the end of the 
day, be part of the funding formula. That is a policy 
decision for our ministers at the end of the day. This 
approach will facilitate the examination of 
alternatives. 


Q17 Mr Wilson: I understand it is less bureaucratic, 
it costs less and will support quality, but, as the 
Chairman said, in Australia that is not what it 
achieved. Lots of research papers have been 
produced, but the overall quality diminished, which 
is why they are switching systems. 

Professor Eastwood: We have made very clear in the 
documents that we need to have a methodology that 
is capable of robust assessment in the manner of the 
RAE. We also commit ourselves in the consultation 
document to model the likely effects of any change 
in the system; and indeed that is one of the things we 
are consulting on as well. We are aware that changes 
in the assessment methodology and changes in the 
funding methodology will lead to changes in 
behaviour. Some of those we might wish to drive; 
some of them we might wish to inhibit. We are aware 
of the issues that you raise. In modelling the shadow 
exercise for 2008, and then in evaluating its likely 
effects, those kinds of questions will be to the fore. 


Q18 Mr Wilson: I have two questions in one, coming 
back to the evidence: what evidence do you have that 
the metrics basis is more efficient than the RAE 
system; and what evidence do you have that the 
metrics basis is less expensive? 

Professor Eastwood: The evidence for the latter, that 
it is less expensive, is that it will enable us to run the 
RAE without the current elaborate infrastructure of 
panels. We think it will also mean that the amount 
of preparation that institutions do for an RAE will 
be diminished. It will not be eliminated but— 


Q19 Mr Wilson: Do you have any estimates of the 
savings? 

Professor Eastwood: No, we have not, but then until 
we have devised the model we will not be in a 
position to— 


Q20 Mr Wilson: It is purely speculation that it will 
be less expensive because you have cut some areas! 
Sir Alan Wilson: Intuition takes us quite a long way. 
I agree with Professor Eastwood that calculations 
will eventually be done, but the present RAE in place 
involves something like 82 different panels or 
committees, with quite substantial numbers of 
members on each, and in a sense there are their own 
administrative demands to go with it. That, alone, is 
taking quite a lot of academic staff time because it is 
the academics that are staffing these 82 panels. We 
are confident that there are gains. What I still feel is 
just as important as any monetary gains which might 
be reinvested are the gains in time. The academics 
that are on these panels are often at the height of 
their research careers, and it will simply free up some 
of their time to go back to the fundamentals of 
research. It is very difficult to measure what the value 
of that would be. 
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Q21 Mr Marsden: Sir Alan, you know very well that 
there have been searching criticisms of the perverse 
incentives that the RAE has produced, not least by 
our predecessor committee in 2001 and by the 
Science and Technology Committee, and indeed 
across the system. Given that this is the case, why 
have you not taken the opportunity of this 
revaluation to do a thorough analysis of the existing 
behavioural impact of the RAE to inform any 
successor system? 

Sir Alan Wilson: 1 think the behavioural impacts 
have been reasonably well understood. The analysis 
of this kind of thing is never perfect, but it is about 
the transfer market. 


Q22 Mr Marsden: Would you like to tell us what 
they are? 

Sir Alan Wilson: The main one which is cited is, as 
it were, the transfer market in academics; the 
possibility that researchers who might have put in a 
certain amount of time in teaching are actually 
negotiating jobs for themselves that are wholly 
research jobs; and that may or may not be a good 
thing. It is very difficult to judge in general whether 
some of the consequences of the existing system are 
good or bad. You might argue that increased 
mobility between universities is a good thing in 
many ways. I think the general feeling is that overall, 
whatever the positive sides, the transfer market is 
too intense at the early stages of an RAE period. To 
my mind, one of the most important things, which, 
again, is terribly difficult to measure, is that the 
RAEs almost have to be organised in terms of 
traditional disciplines. The Funding Council has put 
an enormous amount of effort in trying to add a 
structure which deals with  interdisciplinary 
research; but we are almost certainly moving into an 
era where in science subjects a large proportion of 
the research undertaken will be interdisciplinary. 


Q23 Mr Marsden: I would agree, certainly from my 
own soundings of academics and my own knowledge 
of the area, with all the criticisms of the existing 
RAE that you have enunciated. The point about the 
transfer issue is particularly relevant in humanities 
and particularly with younger academics. However, 
none of that answers the question as to why that case 
is not made systematically here, and how a metrics 
system will improve on that. The criticisms that have 
so far surfaced have not been so much that the RAE 
is absolutely wonderful and should not be touched; 
it is that the RAE has significant flaws, which you 
yourself have underlined today; but there is nothing 
in what is on the table at the moment from you to 
suggest that those significant flaws will be addressed 
by a metrics system, is there? 

Sir Alan Wilson: What we have said in the 
consultation paper is that we are certainly aware of 
the consequence of the RAE as it exists in terms of 
perverse incentives, and any system—because as the 
Chairman said at the outset, it is connected to 
funding—is bound to have incentives relating to 
funding. We have identified in our own minds some 
of the dangers of the new system, an obvious one 
being that if Research Council income plays a major 


part in the funding formula, then research councils 
could be deluged with a much larger number of 
applications for research. 


Q24 Mr Marsden: The fact still remains that what is 
on the table at the moment—which I accept is 
consultation—is a leap from a system which has 
fundamental flaws, and criticisms have been made of 
it, to a system that has not been tested, where we 
have no evidence—this is all intuition’ or 
assumption—that it will not make matters worse. 

Sir Alan Wilson: In terms of the untested nature of 
it, quite a lot of work has been done on correlations 
and so on, as I mentioned earlier. One of the reasons 
why it is very important, and something that has 
been a plank of policy for two years or more—there 
should be a shadow exercise alongside the 2008 
RAE—planned in the light of a consultation which 
is now on the table, will produce the tests, in a sense. 


Q25 Mr Marsden: Are you saying it is a “suck it and 
see” exercise? 

Sir Alan Wilson: 1 hope the consultation and the 
models that have been published with it make clear 
that while there is a reasonably general view among 
many people that a metrics system can replace the 
existing RAE, the tests that have been carried out so 
far show that it is a non-trivial exercise. There are a 
lot of these correlations and a lot of finer points that 
have to be dealt with. What we are trying to do in the 
consultation paper is to expose these so that the 
community can help us make progress. Part of the 
argument about haste, which understandably we 
keep coming back to, Chairman, is that to get this 
apparatus in place following a consultation, to have 
a good shadow exercise, which will be Mr Marsden’s 
test in 2008, means that we are already on a relatively 
tight timetable. I would be reasonably confident that 
the consultation would certainly throw up 
something that we have not thought of at this stage, 
but will get us in a position by around the turn of the 
year where the Funding Council and Professor 
Eastwood can start to plan for the shadow exercise 
in 2008. 

Professor Eastwood: If we had been proceeding in a 
different sort of way, then I think it would have been 
a “suck it and see” approach, and of course that 
would have been wholly inappropriate; but to 
elaborate what Sir Alan has been saying, 2008 will 
run. It will run as a robust exercise, and it will run on 
the basis of what I describe as the Roberts revision, 
and it will be taken forward by the funding councils. 
Alongside that we will then run the shadow exercise. 
We will evaluate that shadow exercise in the context 
of RAE 2008. That will give us the opportunity, on 
the basis of that real-time test of the alternative, to 
make some modifications and do some re- 
engineering, if that is appropriate. That is before it 
is run in earnest. That is an orderly process. At the 
moment, to construct the model that will be run in 
shadow form, then to run that shadow form to 
obtain an appropriate evaluation; then to make the 
amendments which seem appropriate prior to 
running the new model—as Sir Alan says, that 
process, while it might look as if we are moving 
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swiftly now, is a process that we were embarked 
upon already in the ways I was suggesting earlier on, 
with work going on behind the scenes. It is coming 
into the full glare of publicity, but it was actually 
going on behind the scenes. 


Q26 Mr Marsden: Given all of that, how is this going 
to address the issue, which is often raised, that the 
RAE exercise, and for that matter the metrics 
exercise, will merely entrench rewarding research in 
universities that have excellence in them at the 
moment, and not encourage it in others; that you are 
going to create a two-tier university research system? 
Professor Eastwood: That is where the capacity of 
the existing methodology and perhaps an enhanced 
capacity in the future—a methodology to identify 
excellence—is important. The distribution of QR, 
the form of funding that comes from the RAE, is 
broader across the sector and broader across 
institutions than the distribution of any other form 
of research funding; and there is nothing in what we 
are proposing which would disturb that. 

Stephen Williams: We have heard the Roberts review 
mentioned a few times. That was in 2003, and all 
universities are currently, so they have told me, in 
advanced stages of preparation for the 2008 RAE. 
Sir Alan, why has the Government decided that now 
is the right time to drop the existing RAE system? 


Q27 Chairman: Stephen is a Bristol Member! 
Sir Alan Wilson: The Vice-Chancellor of Bristol 
University has had well-known views on the RAE, 


much quoted! Having said that, I have now lost: 


track of the question! 


Q28 Stephen Williams: Why now? 

Sir Alan Wilson: 1 was going to say that, as I 
indicated earlier, it is not so much now. It goes back 
to the 2004 10-year science investment paper that 
was published under the logos of three government 
departments, and it was assumed then that there was 
a strong presumption that something like a metrics 
basis would take over after the 2008 RAE. It is not 
that this has just been thought of in a very short 
space of time; it goes back at least two years. In terms 
of earlier reviews, they are ideas that have been 
around for a long time. The particular issue now is 
really what I indicated earlier about trying to get a 
good shadow exercise in place by the time of the 
2008 RAE. 


Q29 Stephen Williams: So when all these reports that 
we would have read after the budget speech back in 
March, after the Chancellor’s announcement about 
people expressing surprise—all those people should 
have seen it coming in 2004. 

Sir Alan Wilson: | am sure they did see it coming in 
2004, Chairman; it was very clearly there. I think 
inevitably when there is a publication it brings it to 
the top of people’s minds and potentially affects 
funding, and people start to think about their future 
strategies. I think that that is very understandable. 


Q30 Stephen Williams: Sir Alan, did your 
Department request this review, or was it acceded to 
a request from another department? 

Sir Alan Wilson: 1 do not think anybody has been 
acceded to anybody else; it has been a process that 
has been in place since the 2004 10-year framework. 
We jointly publish annual reviews of that. We 
published one, and I think the second one is about 
to be published. I think it was convenient, and 
important indeed to use the framework in the budget 
to draw a number of threads from that 10-year 
framework together in 2006, and research clearly 
had to be part of that. 


Q31 Stephen Williams: Chairman, there is some 
suspicion, as both witnesses will have heard from 
earlier questions, about which part of the 
Government is driving the process; whether it is HM 
Treasury, which has the primary role, or the DfES. 
If you look at the timetable, it is all built in from the 
stop point, which appears to be the Chancellor’s pre- 
budget report, which traditionally is in November in 
this country, feeding back to the end of consultation 
at the end of October. The consultation has just 
started. Has this whole process not been truncated 
just so that the Chancellor can have the answer he 
wants by the time he comes to his pre-budget report 
in November? 

Sir Alan Wilson: | do not think it is about the 
Chancellor’s timing, Chairman. The fact that 
Professor Eastwood and I are sitting here before you 
now as co-chairs of the group that produced the 
consultation paper is a demonstration that DfES— 
and in its relationship with the Funding Council— 
is one of the lead partners in this exercise. We have 
certainly had representatives of HM Treasury and 
the DTI on our consultation group, and indeed the 
other funding councils. What has come out has been 
agreed by all parties. 


Q32 Stephen Williams: Is it the DfES’s consultation 
rather than jointly with the Higher Education 
Funding Council? 

Sir Alan Wilson: It is essentially joint. Everyone has 
an interest in it. All the Ministers of the relevant 
departments have an interest in the policy 
developments. We have indicated areas like applied 
research and so on, which will be major policy 
questions in years to come. The implementation of 
the outcome of this consultation, the shadow 
exercise, will be the responsibility of the Funding 
Council. I think that is why certainly from the DfES 
point of view it is very important that Professor 
Eastwood is closely associated with this. 


Q33 Stephen Williams: Professor Eastwood, your 
predecessor from your new forthcoming role, Sir 
Howard Newby, has said that policy goals should 
come first rather than the detailed consultation on 
the types of metrics; that we need a review looking at 
what the UK needs from higher education research. 
He said that running a metrics review is no substitute 
for a higher education policy review. Do you agree 
with that? 
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Professor Eastwood: 1 think the expectations of 
higher education in this sphere are well known and 
well articulated. The 10-year framework is one 
articulation of that; there was also an articulation of 
that in the HE White Paper back in 2003. I think that 
higher education is well aware of its responsibility to 
maintain the UK’s position as number two in the 
international league table for research performance. 
I think the emphasis on translation of research is one 
that universities are well aware of, and indeed 
sympathetic to. The importance of maintaining 
funding for blue-skies research is right at the heart of 
the dual-support system. In that sense, the key policy 
drivers are very clearly articulated. 


Q34 Chairman: But are they, Professor Eastwood? 
We have still been trying to get to the heart of the 
policy drivers here. Is it saving money; is it better 
allocation of research resources across our country; 
is it international competitiveness? I am still not sure 
what is at the heart of the policy drivers. 

Professor Eastwood: | think it is all of those, and it 
is also having an assessment methodology and a 
funding methodology that is fit for purpose for the 
next 10 or 15 years. 


Q35 Chairman: It takes senior academics out of the 
loop! 

Professor Eastwood: | do not think, with respect, 
that that is what is being proposed. 


Q36 Chairman: Sir Alan was looking quite pleased; 
that all these academics are going to be released to 
do more research and not going to be involved in 
the loop. 

Professor Eastwood: Sir Alan can speak for himself 
of course, but there is a distinction between 
diminishing the weight of peer review, which is there 
in the current methodology, which is what Sir Alan 
was talking about I think, and maintaining the 
confidence of the sector in the system of assessment 
of funding that we evolve. That confidence will be 
there partly because the methodology will have been 
demonstrated to be fit for purpose, and partly 
because peer review does sit behind a number of the 
income metrics that we will be using. Research 
councils use very heavy peer-review for example. 
There is a suggestion in the consultation document 
that there be an overall assessment of institutions’ 
research plans and research directions. I think that 
in different ways there will be those kinds of inputs. 
The issue around metrics is not about a move 
towards an utterly desiccated system, but a move 
towards using some proxy indicators for quality. 


Q37 Stephen Williams: Amongst Professor 
Eastwood’s list of reasons for doing it was to 
maintain the UK’s competitive position as number 
two in the world after the US in world research 
rankings. Is not one of the advantages of the current 
system that a vice-chancellor, whether from UEA or 
Bristol, can go to China or somewhere else in the 
world and say, “I have got this 5-star department” in 
whatever it is, and everywhere in the world people 
will recognise that? They will also understand the 


system by which you have arrived at that assessment. 
Are you not at risk of jeopardising that world-wide 
confidence in what we have at the moment? 
Professor Eastwood: | absolutely share with you a 
sense of the importance of that ability to badge 
quality in the UK. There is no doubt about that, and 
I tried to refer to that in some of my earlier remarks. 
I would just make two supplementary points. One 1s 
that RAE 2008 will articulate quality in a different 
way, and so will not have 5 and S5-stars; it will have 
a graded profile. That was one of the Roberts 
responses to a previous criticism of the RAE and in 
particular the funding that flowed from it. That is to 
say it has a kind of cliff edge, and if you were 5, 5- 
star and then 4, there was a big fall; so the new 
methodology is designed to smooth that. There will 
be a different language of excellence emerging from 
RAE 2008. I do think it is vital, and the consultation 
believes it is vital that the new system should be able 
to identify, to describe and to badge research 
quality, for precisely the reasons you have given. 
Sir Alan Wilson: \t is interesting in the context of this 
discussion that the analysis that establish this 
country as number two in the league table and 
second only to the United States is essentially a 
bibliometric analysis, an analysis of papers 
published in different kinds of journals, and then the 
number of citations—in other words, the extent to 
which this research has actually been used. That has 
been one of the main measures that has established 
the league table in which the communities have some 
confidence. The problem with metrics is that they do 
not cover the full range of subjects; but it is a field 
that is developing very rapidly, and this kind of 
international benchmarking—I think we are 
beginning to be reasonably confident—can be 
addressed through the use of those kinds of metrics. 
We are not just talking about research income; we 
are talking about available measures on a global 
basis of the value of research, as perceived through 
citations of published work. 

Chairman: Do you want to drill down on this metrics 
system—David? 


Q38 Mr Chaytor: Why are the Australians 
abandoning the system that we are now moving 
towards adopting? 

Sir Alan Wilson: In trying to answer the question, I 
cannot claim familiarity with the Australian metrics 
system—perhaps I ought to be able to! My guess is 
that the position that we have reached in this 
country has needed 20 years of the research 
assessment exercise to produce a foundation and 
move the university system in this country from 
where it was in 1986 in research terms to where it is 
now. I have no doubt that in that sense it has been 
very valuable. However, if I can make another 
personal statement, I believe that diminishing 
returns have set in, and there is a danger that 
particularly the failure to address the 
interdisciplinary agenda and to take on the biggest 
challenges because there is risk—and if you want to 
make sure you have your four publications for an 
RAE panel, you perhaps take less risk than at least 
we would want some of the community to take on. 
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Q39 Mr Chaytor: What happens in the United 
States, or is the system so different that comparisons 
cannot be drawn? 

Sir Alan Wilson: It is essentially very different. You 
have a variety of funders. You have a different 
system for contributing overheads from government 
funding in particular. By that I mean the research 
funding through National Science Foundation and 
National Institute of Health and so on. It is not easy 
to draw comparisons. 


Q40 Mr Chaytor: On the arguments in favour of the 
metrics approach, you have cited costs, saving of 
time and diminishing returns and the changing 
nature of research and the growth of 
interdisciplinary research—but how exactly will 
some of the metrics criteria listed in the consultation 
document respond to the interdisciplinary issue 
more effectively? For example, how will the 
bibliometric approach reward interdisciplinary 
studies more effectively? 

Sir Alan Wilson: Can J add to that research income 
of various kinds, and take the two kinds of 
indicators? In relation to research income, research 
councils are increasingly funding interdisciplinary 
projects and those defined on an interdisciplinary 
basis rather than on a traditional subject basis. That 
will be reflected in research income so that 
universities that are more successful in winning 
funding for interdisciplinary projects from research 
councils— 


Q41 Mr Chaytor: Why can that not be adequately 
reflected by a process of peer review? 

Sir Alan Wilson: My own instinct about this is that 
there is a massive combinatorial problem. There are 
too many combinations. If you look within the 
sciences particularly at the combinations of 
disciplines that produce an_ interdisciplinary 
research project, if you wanted to find a panel for 
something like bioinformatics, which should be a 
combination of chemistry, biology and computer 
science, you would need a very large number of those 
panels; whereas experts in a field within a research 
council can find ways of doing that. If we look at the 
aggregate of all those— 


Q42 Mr Chaytor: At the moment presumably there 
are a large number of single discipline panels, but 
how many panels are there at the moment? 

Sir Alan Wilson: There are 67 subject panels and 15 
so-called super panels. 


Q43 Mr Chaytor: Is it not a question of reducing the 
number of 67 down to a smaller number of broader 
interdisciplinary panels? Why would that not work? 
Sir Alan Wilson: | think what happens when you try 
to do that is that you always have a good number of 
cross-boundary flows, however you try to define that 
smaller number of panels. What we also have to bear 
in mind, Chairman, is that in the consultation 
document we have kept open the question of the 
possibility of having exactly those kinds of broader 
panels to take an overview of the metrics system that 
is developed, because obviously something which is 


entirely mechanism certainly in its early days could 
have dangers. I have not responded to Mr Chaytor’s 
question, Chairman, on bibliometrics, which is 
another side of the metrics analysis. It is very 
interesting on a global basis: there are an increasing 
number of interdisciplinary journals, so if you are 
looking at successive citations from journals, the 
publishing world has responded to _ the 
interdisciplinary agenda very rapidly, and so it gives 
us a base which reflects the research frontier rather 
more accurately than traditional subject panels 
would. 


Q44 Mr Chaytor: Would that argument apply 
equally to the other criteria that are listed on the 
consultation documents, simply the raw numbers of 
research active staff or the number of some PhD 
students? You are arguing that increasingly research 
active staff would be engaged on interdisciplinary 
work, the numbers of research students would be 
engaged on interdisciplinary work and— 

Sir Alan Wilson: 1 think, Chairman, it is Mr 
Chaytor’s question about panels turned into the way 
the metrics are used in a formula, how broad are the 
groupings and that is something that has to be 
explored, so in the models that we presented and put 
on the web site for illustrations we have actually 
looked at different kinds of groupings, but I cannot 
say at this stage that we know what the final 
recommended answer will be and that is why we 
have actually put alternatives on the website to 
inform the consultation. 


Q45 Mr Chaytor: In terms of the broader issue 
between the role of metrics and the role of peer 
review, do you think it is absolutely an either/or 
choice to be made? 

Sir Alan Wilson: \t is not an either/or, firstly in the 
sense that Professor Eastwood referred to in the 
context of another question and that is lying behind 
metrics, whether it is bibliometrics or research 
council income, there are different peer review 
processes, so it is not as though these numbers are 
not, as it were, products of peer review, but it is a 
different kind of peer review, so in that sense we are 
still connected and, as I said earlier, it is possible that 
certainly in some areas particularly arts and 
humanities and perhaps in social sciences we still 
need panels to work with the metrics base. 


Q46 Mr Chaytor: What would be some of the 
disadvantages of moving into largely a bibliometric 
based system or a largely research income based 
system, what would you identify as the biggest 
problems? 

Sir Alan Wilson: 1 think at the moment I do not see 
huge problems. I mean I think it has to be put to the 
test in 2008 and I think we will learn a lot from that 
exercise. I mean all the signs are, because the 
correlations are so high and if I am right about the 
fact that many of these metrics are closer to the 
research frontier than traditional subject panels 
might be, then I think on balance it is more likely to 
be beneficial than not, but I think it is very 
important, for instance in terms of perverse 
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incentives, we have recommended, or we have put in 
a question since we are not convinced that we will 
have seen at this stage all the possible perverse 
incentives that this has to be monitored very 
carefully. We know what has happened in the past, 
if something goes wrong, as it were, with this one in 
terms of incentives— 


Q47 Mr Chaytor: In terms of incentivising 
behaviour and rewarding behaviour or discouraging 
behaviour, there must be some underlying 
assumption about the kind of behaviour the 
Government wants to encourage and discourage, so 
what do you imagine the outcome will be if post- 
2008 there is a shift to a system that is significantly 
based on metrics and what kind of behaviours will 
be encouraged and discouraged? 

Sir Alan Wilson: | mean in one sense I do not think 
there will be a huge change in that, I mean it is not 
to say that it will not be possible for new entrants to 
emerge more easily than it may be the case in the 
present system, but what we have succeeded in doing 
in this country is generating a community of top 
class researchers and I think those top class 
researchers will still present themselves through 
whatever kind of metrics are used. Now many of 
those are now working in interdisciplinary teams, 
but I suspect if you went back 10 years they were still 
working, they were beginning to work in 
interdisciplinary teams, but they were presenting 
themselves to subject panels and I am sure subject 
panels have done their best in assessing the quality 
of interdisciplinary work, but I think those are the 
kinds of shifts that we will find. 


Q48 Mr Chaytor: In terms of the distribution of 
research funding this must inevitably lead to a 
further concentration? 

Sir Alan Wilson: \ think that is actually certainly not 
the case in terms of the word “inevitably”. 


Q49 Mr Chaytor: Is it likely to lead to further 
concentration? 

Sir Alan Wilson: 1 think if you actually had the 
patience, and it probably does need patience to look 
at the large numbers of tables on the models that we 
have used for illustrations on the web site— 


Q50 Mr Chaytor: I think I am actually going to give 
that a miss. 

Sir Alan Wilson: Probably more of them have moved 
away from increased concentration and in fact there 
has probably been slightly less concentration, if my 
memory serves me correctly, so it is certainly not 
inevitable. 


Q51 Mr Chaytor: Slightly less concentration, 
because we still have the RAE system and a peer 
review system. What I am trying to get at, is there not 
a correlation between those who are in favour of 
greater emphasis on a metric based system and those 
who are in favour of a greater concentration of 
research funding? 


Sir Alan Wilson: \s there a correlation? 


Q52 Mr Chaytor: Is there a correlation between 
the two? 

Sir Alan Wilson: Yes, | mean I am sure there is a 
correlation between them but, at the end of the day, 
it is a policy question for Professor Eastwood and his 
council colleagues because it is the weightings and 
the funding formula as much as the assessment and 
the metrics that will actually determine the degree of 
concentration. Whether it is metrics or RAE neither 
of the systems determines the degree of 
concentration, it is the weightings that go into the 
funding formula. 


Q53 Chairman: Sir Alan, you know a famous 
scientist, Joe Schumpeter said “politics about who 
gets what, when and how” and two and a half years 
ago when we looked at higher education one of the 
things that really worried us was a very well known 
vice chancellor came into this room and said he 
believed that a much greater concentration of 
research in a handful of research, which at university 
is a handful, and I said, “Do you actually mean five 
only?” and he said, “Yes, five”, and our report said 
that would mean that most of the research in 
research funding universities would be in London 
and the South East, it would cut out Leeds, your old 
university, it would cut out your university, the 
University of East Anglia and we would have that 
concentration and then what we said is if there was 
nota research rich-science base in each of the regions 
of our country, it would be a very retrograde step. 
Are you telling me that there is not something going 
on here, whether you call it metrics or the changing 
system, that actually is not the agenda of a small 
group who still want—they are very articulate, they 
are not secret about this, Sir Alan—who want 
research based to be based on a handful of 
universities; come on? 

Sir Alan Wilson: 1 fully understand the question, 
Chairman, and I would say very clearly that there is 
no hidden agenda of that kind. 


Q54 Chairman: Not from you, no. 
Sir Alan Wilson: The policy of the Government is to 
fund the best research wherever it is found. 


Q55 Chairman: Even if they ended up in five 
universities? 

Sir Alan Wilson: | think that the system from where 
we now stand, you know, it would take a very 
different kind of policy change to concentrate 
research funding in five universities. No RAE or 
metrics based system of any kind that we have 
thought about would bring about that kind of 
concentration, it would need a very different kind of 
policy directive and that, to the best of my 
knowledge, is not under consideration by any of our 
ministers. 
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Q56 Chairman: It is worth us bearing in mind what 
major players are saying? 

Sir Alan Wilson: Perhaps I could add, Chairman, 
that what I am arguing is in part connected to what 
has been a core principle for a very long time which 
is the best research should be funded wherever it is. 
You can then apply different levels of some 
selectivity, as it were, and still mattain that 
principle, but I think any level of selectivity, and in 
a sense I am repeating myself, you know would not 
generate research in five universities, it would need 
somebody to say, you know, “we will just fund 
research in five universities”. I think the other thing, 
Chairman, is what you have said yourself about the 
regional dimension and as a government department 
speaking as someone from DfES I mean we are very 
well aware of PSA _ targets for economic 
development across the country and we are very well 
aware of the contribution that universities make to 
that agenda and so we actually have discussions, 
whether it is with HM Treasury or with CLG or DTI 
about how HEs contribute to that and particularly 
the research agenda, so we would want to support, I 
am sure Chairman, research across the country. 
Professor Eastwood: If 1 could just add that the 
concentration of the kind that you were sketching 
would run counter to the considered position of the 
funding council, would run counter to the view, I 
think, of the research councils, both of whom see 
distributed excellence, if I can use that term, as 
fundamental to maintain the supply of outstanding 
researchers as well as meeting the regional case that 
you have articulated. I think it is also worth noting 
that alongside the recurrent investments we have 
been talking about when we have been talking about 
QR or the research councils, there has been a big 
commitment to invest in the research infrastructure 
in general and the science research infrastructure in 
particular and if you look at the pattern of that 
investment through initiatives such as SRIF, that 
investment too in pretty heavy kit, again is 
distributed over a substantial number of 
institutions, so it seems to me that there is, as it were, 
a broad consensus around the way in which we are 
investing both in capital terms and in recurrent 
terms in the research base which reflects the funding 
of excellence, selectivity, but appropriate 
distribution. 


Q57 Chairman: | still doubt whether out of this 
Committee and I have got a restricted group of 
people in this country, they really understand; a lot 
of people out there understand the research 
assessment exercise, peer reviewed the system 
committees, however many there are, and they 
understand that system works. Now this system 
called metrics people out there I do not think, and I 
bounced this question over the weekend, “What do 
you mean, what are you measuring? 
Mathematicians—what are they going to measure 
then and how do they know those measurements are 
fair? What is this metrics?”. 

Professor Eastwood: It is certainly not a good dinner 
party discussion topic. 


Q58 Chairman: No it is not. Ordinary people in this 
country ought to be able to understand, it is 
assumably camouflaged by some _ sort of 
nomenclature that is impenetrable. What are you 
measuring? 

Professor Eastwood: What we are measuring is the 
investment that is going into research through 
income measures, we are measuring the outputs in 
terms of the growing use of bibliometrics and the 
impact that they have and we are measuring volume 
in terms of the number of researchers, the number of 
PhD students and so forth. A number of those things 
have been there in the RAE before, they are not new, 
and a number of the peer review panels, particularly 
in the subject areas which were the first to develop 
bibliometrics, used those, and the economists would 
be a good example, used those as an aid to make 
their judgments about the quality of published 
outputs in 2001 and, to some extent even in 1996, 
and will do so again with the Funding Council’s 
blessing in 2008, so metrics have been there and I 
think if you look at some of the indicators, notably 
bibliometrics, what they are telling us is now much 
more sophisticated than it was even five years ago, 
they are telling us very interesting things about the 
nature of impact of published research, when it has 
an impact, where it has an impact and it seems to me 
that what we are offering in 2008 is the possibility of 
a very serious evaluation of the sophistication of 
those sorts of metrics. 


Q59 Chairman: So all this, Professor Eastwood, 
much more sophisticated measurement, all these 
measures go into, and who makes the decision about 
allocation, because that is not neutral, is it? Does it 
go into a wonderful computer that says, “This is the 
distribution of resources for research”, or does it go 
to human beings in committees? 

Professor Eastwood: Ultimately the responsibility 
for the distribution of research funding to 
institutions is statutorily the responsibility of the 
Funding Council. In coming to that judgment the 
Funding Council obviously is aware of the policy 
framework and is responsive to the _ policy 
framework and, as Sir Alan was saying earlier, and 
it is clear in the published consultation, that at an 
appropriate level there will be a number of panels to 
advise on the appropriateness of the metrics. 


Q60 Chairman: It is a big shift away from decision- 
making from academics to bureaucrats and quangos 
like yourself. That is the shift of power, is it not, you 
are taking away from academics, you are giving 
them much more time to play with their research, 
but the decisions will be made by people like you 
who really will jump to what the Government wants? 
Professor Eastwood: | think that is a very stark 
characterisation and not one I would necessarily 
agree with. I think the first thing to say is— 


Q61 Chairman: Professor Eastwood, I have been in 
this chair for quite some time and I am used to 
dealing with people in your job and I have to tell you 
that it is very different, if HEFCE does something, a 
whole bunch of academics who are independent .. . 
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Once you become a paid employee of HEFCE, you 
do become a very different kind of person, do you 
not, you must realise that? 

Professor Eastwood: Perhaps about next year I will 
come and comment further on that. 


Q62 Chairman: It is interesting, is it not? On a 
serious point, Professor Eastwood, the sort of thing 
you have been telling this Committee today is not 
what Howard Newby was telling the Royal Society 
only 10 days or two weeks ago when I chaired the 
meeting over there and it is certainly not what 
Gareth Roberts said either, so the view there is some 
kind of unanimity, your predecessor at HEFCE is 
not singing from your hymn sheet today and Gareth 
Roberts is not singing from yours, Sir Alan, that is 
the truth of it, is it not? You have come here, all 
agreed, everyone out there, all reasonable men and 
women agree with us today; you know that is not 
true? 

Sir Alan Wilson: Chairman, as I said to Mr Wilson, 
it is a highly controversial business and I think we 
can all agree about that, but that goes both ways 
round, it is not the successive proposals against 
traditional RAE, it is very complicated, but I think 
what I would argue, Chairman, and it goes back to 
your comment again about taking academics out of 
the loop, I mean I think what in a sense we are trying 
to do is to put the academics back into another and 
more productive loop in terms of research. There 
will still be bidding for funding from research 
councils, there will still be peer review there and 
indeed some of them, Chairman, will still be on 
Professor Eastwood’s council so it is not simply, as 
it were, officials deciding what happens to the 
funding. I think what is important, and I go back to 
your earlier point on this, is that whatever we do and 
whatever the system produces in the end, is 
transparent and understandable and I do not think 
this has to go into obscure mathematics and in fact 
if you actually take the RAE in its present form, I 
think Professor Eastwood has been hinting at this, 
there is some subjective academic judgment in 
actually producing research ratings, getting from 
those research ratings and then starting to use 
metrics, getting from those research ratings to a 
funding formula involves the same kind of 
complications, whether it is in the RAE framework 
or whether it is in the metrics framework and I think 
we do have a responsibility between us to explain. 


Q63 Chairman: Sir Alan, you know I am a great 
admirer of yours and of Professor Eastwood, but it 
is my job as Chairman to this Committee to tease out 
some of these commonsense things that people will 
ask and they do ask the thing, “Is this movement”, 
like the Learning and Skills Council in further 
education, “becoming the all powerful body?” Here 
in higher education are we seeing a transfer from a 
whole group of committees and academics making 
the decisions to a bureaucracy, wherever nicely 
described, the civil service, in your case Sir Alan, and 
a quango in your case Professor Eastwood, is that a 
shift of power and should people be worried about 
it? Perhaps they should not be worried about it. 


Professor Eastwood: | think in terms of the process 
that we have set out, the first serious commentary on 
the alternative will come from the shadow metrics 
exercise in 2008, it will come from our panels, so the 
first group to offer substantial commentary will be 
academics. The consultation suggests that there 
might be a different approach in STEM subjects 
from the arts and humanities and some of the non- 
quantitative social sciences and in the areas where 
we think metrics might be more robust and that will 
be tested in 2008, there will be panels to advise, so I 
think it is a process which might dial out some of the 
complexity and we hope might dial in some 
transparency, but I do not think it is a process which 
can be described, as it were, simply leaving the 
funding to apparatchiks. On the contrary, the 
judgments which underpin the metrics are all peer 
review judgments, whether it is the distribution of 
funding on the one hand or the decision to cite an 
article or research output on the other, so academics 
are absolutely at the heart of this and what we are 
trying to do is to find the most efficient way of 
deriving quality judgments from those data, 
ensuring that they are valid and validated and then 
the Funding Council, in a way which is appropriate, 
will translate that into funding outcomes as it has 
done since the 1980s. 


Q64 Mr Marsden: What I would like to do now, if I 
may Chairman, is to move on to some of the specific 
impacts of these changes both in the so-called STEM 
science technology, engineering and maths subjects 
and also in arts and humanities. Just to make the 
observation in passing to you, Professor Eastwood, 
and you talked about disputed excellence, but there 
is also of course the law of unintended consequence 
and I do not think anyone on this Committee doubts 
your bona fides in that respect, but we are trying to 
examine what some of those unintended 
consequences might be. The first question I have got 
is for you and it is very much focussed on the issue 
of arts and humanities. How can a metric system in 
any shape or form, however modified, replace the 
RAE when a metrics based system cannot deal with 
arts and humanities subjects which account for 
about half of the existing research activity? 
Professor Eastwood: The first thing to say in 
response to that is that HEFCE and the AHRC— 
Arts and Humanities Research Council—have set 
up a group to advise precisely on that and the view 
interestingly of the AHRC is that it is possible with 
or without some elements of traditional peer 
review— 


Q65 Mr Marsden: It is possible? 
Professor Eastwood: With or without some, and this 
will be determined, some elements of— 


Q66 Mr Marsden: They are not ruling out right from 
the beginning metrics as an ingredient? 

Professor Eastwood: No, indeed they are not, and 
indeed the humanities’ panels in 2008 will be using 
some metrics because all panels in 2008 use at least 
15%, or at least 15% of their judgment is metrics 
based, so metrics are there and what this group 
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chaired by Professor Michael Wharton of UCL will 
do is evaluate the way in which metrics, particularly 
bibliometrics, have developed in the humanities and 
feed in to the consultation in the autumn its 
judgments. 


Q67 Mr Marsden: Can I just ask you, I mean in 
terms of we toss this phrase “bibliometrics” around, 
but I know from a previous existence that journal 
articles carry different weights depending on the 
journals in which they appear; a very specialist 
journal article might attract a particular weighting, 
a more generalist journal article would attract a 
broader rating. Is that going to be spelt out, made 
more transparent under a metric system, a 
bibliometric system, any more than it is under the 
RAE system? 

Professor Eastwood: You mean the weightiness that 
particular journals might carry? 


Q68 Mr Marsden: Yes. 

Professor Eastwood: | think we can already see it in 
some areas of the social sciences and actually in 
some areas of the humanities that there are 
particular journals which carry particular weight, 
particular journals that academics are desirous of 
publishing in. 


Q69 Mr Marsden: And is that, to come back to the 
original point that I raised, is that transparent in the 
same way as we have league tables for schools, are 
you envisaging a situation where anyone could go 
along and say, “Professor X has contributed to that 
department because he got three points from”, I do 
not know, “his New Scientist article as opposed to 
two points from a more specialist article on 
biochemistry”, for example? 

Professor Eastwood: 1 do not think that one 
precludes the other because I think it is in the end 
about quality. I think the issue for the arts and 
humanities in particular is how far they are down 
that particular road and the kind of bibliometrics 
that are now increasingly available. For example, 
there are metrics around monographs becoming 
available for the first time which is obviously of a 
very considerable importance to the humanities. 
Having set up this group to advise, I do not want to 
pre-judge the outcomes. 


Q70 Mr Marsden: I am not asking you to pre-judge 
it, 1am asking you to try and elucidate what is quite 
an Opaque subject, even the term “metrics” is an 
Opaque one and to try and translate what it would 
mean for your average academic or your average 
university department in, for instance, arts and 
humanities under a new system. I would like to ask 
you another question in that context and that is to 
say, supposing I am a young academic in my mid to 
late thirties and I am trying to decide, I want todoa 
big book, it might be a big history book or it might 
be a big book on physics or whatever, you know as 
well as I do that one of the criticisms of the present 
system is that the RAE forces that young academic 
to do chopped-up little articles in rather narrow 
journals rather than having the time to pursue bigger 


issues in a bigger book. Is a metric based system 
likely to, or a system that takes in metrics, likely to 
make that easier or more difficult for someone in the 
arts and humanities? 

Professor Eastwood: If the conclusion was that 
metrics can measure the impact and quality of 
monographs, which is a claim which is now being 
made, then if there was a problem there it will start 
to address that problem. My own view is that some 
of the effects of the RAE in this regard have been 
exaggerated, particularly in a seven year cycle it is 
perfectly possible to produce a monograph to 
produce major articles and even to produce articles 
for noble publications such as History Today. 


Q71 Mr Marsden: This is not what many of your 
historian colleagues say, people in the Royal 
Historical Society, the Institute of Historical 
Research and in other organisations who are 
concerned that the current system in RAE is 
stopping some of those big books being produced by 
younger academics, let alone a more metrics based 
one? 

Professor Eastwood: 1 think I would say with the 
greatest of respect to my colleagues that— 


Q72 Mr Marsden: You acknowledge people are 
saying this? 

Professor Eastwood: | acknowledge people are 
saying this, but I think it is a misdiagnosis, I think 
there are a number of wider trends which have little 
to do with the RAE and that are attributed to the 
RAE and what we are seeing, for example, in 
monograph publications, something which is 
happening in publishing, and the difficulties of 
getting monographs published, is much less an 
artefact of the RAE than an artefact of the 
publishing industry and the information revolution. 


Q73 Mr Marsden: Let me turn to the STEM subjects 
then. Do you accept, Sir Alan, that it is going to be 
a lot easier to produce a metrics system based on 
STEM subjects than arts and humanities? 

Sir Alan Wilson: It is certainly true at the present 
time, partly so because of the volume of research 
council and other research funding, so you 
potentially get more sensitive indicators and hence 
the Annex A to our consultation paper where we 
explore a wider range of indicators and input on arts 
and humanities. 


Q74 Mr Marsden: You said yourself earlier on that 
you had been influenced by the concerns that had 
been expressed about the current RAE system 
perhaps inhibiting collaborative and disciplinary 
research costs and STEM subjects on a number of 
different departments. Are you convinced that that 
particular issue, which is a real issue, is one that 
would be addressed by a predominantly metrics 
approach in those subjects? 

Sir Alan Wilson: | think it makes it a lot easier. I 
think research councils and other research funders 
are putting an increasing proportion of their funding 
into interdisciplinary projects and that would be 
represented in a funding formula. 
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Q75 Mr Marsden: What would you say to those 
people who say that these proposals have essentially 
been drafted, or this consultation has essentially 
been drafted, with a focus on STEM subjects and 
that the arts and humanities have really been a 
tagged-on after-thought? 

Sir Alan Wilson: Well it is certainly not an after- 
thought, Chairman, and again hence the annex. I 
mean I think we wanted to produce a comprehensive 
response. I think perhaps I could make one related 
comment that builds on the big book question or the 
individual academic question. I mean one of the 
things that we have said is I think there is a danger 
in thinking that we are producing a system that 
focuses on individuals. It will actually operate at a 
much more aggregate level and what we have said in 
the consultation paper is that we believe that 
universities are now increasingly in a position where, 
because of past RAEs, they now have to manage 
their research internally and I think the young 
historian who wants to produce a big book, in that 
case I would be disappointed if the university 
management in either the department or the 
university was not, in effect, supporting that and I 
think that is one of the challenges. 


Q76 Mr Marsden: I would love to believe that and 
anecdotal evidence suggests in a sense that it does 
not always happen. Can I ask you, Professor 
Eastwood, a final question? When you were being 
pressed earlier about the _ possibilities of 
measurement you referred to “input” and “output”, 
you said, “We want to be able to measure input, we 
also want to measure output”. I mean output surely 
in terms of research also involves the impact of that 
research; I mean obviously it may involve something 
very directly like a scientific application, it also is an 
impact on various groups of people. Does not the 
impact of research also involve the impact on 
teaching and dissemination? 

Professor Eastwood: Yes, | think it does. 


Q77 Mr Marsden: In which case why are you not, or 
maybe you are going to, why are you not going to 
use the opportunity of this consultation to look at 
the impact of research on teaching and how you 
would measure that either within the existing RAE 
context or the metrics context? 

Professor Eastwood: | think I would make two 
comments there: one is that through a different 
initiative there is now a funding stream available to 
ensure that those institutions which have less QR, 
nevertheless— 


Q78 Mr Marsden: But a.much smaller amount of 
money? 

Professor Eastwood: It is a smaller scheme, but 
nevertheless it goes in the direction of the point that 
you are making which is that in higher education, the 
teaching should come out of the research an 
informed environment, so that is the first point I 
would make. The second point I would make is in 
the current RAE, particularly in the humanities 
where what one has are, if you like, publications 
which represent a synthesis and the creativity is at 


the level of that synthesis, then they are publications 
which, quite appropriately, are assessed within the 
RAE. I think there is a deeper issue which we may 
return to on another occasion no doubt which is 
given that we might shift the focus to teaching, what 
is the appropriate funding for teaching which will 
facilitate that kind of research and teach, but that is 
a related, but I think a distinct question. 

Jeff Ennis: My first question, Professor Eastwood, is 
really a follow-on to the line of questioning that both 
the Chairman and Gordon has just been pursuing 
recently in terms of the utterances from Sir Howard 
Newby in terms of the way we are approaching this 
issue, because he said on RAE at the HEPI 
conference on 21 June that the order of any 
discussion must be to clarify policy goals first, then 
consider the RAE and then consider metrics—not to 
begin with a consultation on metrics which will 
impact on the RAE which will impact on research 
policy and we have obviously had organisations 
such as Universities UK, to name but one, who have 
been calling for a full debate on this issue. How 
would you answer those particular points? 


Q79 Chairman: When do you take over Professor 
Eastwood? 
Professor Eastwood: | take over in September. 


Q80 Chairman: Beginning of September? 
Professor Eastwood: September 1, yes. 


Q81 Chairman: Sir Alan, are you moving out of your 
present job? 

Sir Alan Wilson: 1 am afraid Professor Eastwood 
and I only overlap by one month because I will leave 
on 30 September. 


Q82 Chairman: And you are going to become 
Master of Corpus Christi in Cambridge? 
Sir Alan Wilson: Indeed. 


Q83 Chairman: Congratulations on that. Do we 
know who your successor will be? 

Sir Alan Wilson: Not yet. 

Professor Eastwood: | think it is wrong to see the 
consultation document in isolation, I think the 
consultation document has to be contextualised. 
Next Steps is a part of the immediate context for 
that, but the 10-year strategy is there, as is HEFCE’s 
own current corporate plan, so I think that the issue 
that Sir Howard rightly pointed to—is there a policy 
analysis, are the principles articulated? Yes, they are, 
they are in the 10-year framework, they are in our 
current corporate plan. Is what is being consulted on 
running counter to those? No, I do not think it is. 


Q84 Jeff Ennis: I am trying to work out why an 
organisation like Universities UK would say 
something like that or Sir Howard Newby. Might 
they think that the Government is working to a 
hidden agenda here do you think or— 

Sir Alan Wilson: Can 1 comment on that and it 
connects to one of your earlier comments, 
Chairman, about academics being taken out of the 
loop which resonates very strongly with me? I mean 
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I am rather surprised that UUK would, as it were, 
call for a full debate, because there is a sense in which 
the availability of the full debate is there almost 
continually. I would say the Department through its 
ministers and through its officials and indeed the 
Funding Council have a pretty good record of, as it 
were, continual engagement with the sector and 
continuing consultation as is clear in the central 
discussion we have been having and very fortunately 
academics do not hold back, so if they actually have 
comments to make about government policy then 
we hear them. I actually do not feel we are in a 
vacuum. I agree with Professor Eastwood that there 
is a policy framework within which this is set and I 
do not think from the nature of our normal 
engagement with the sector that we are cutting off 
any debate and there is certainly no hidden agenda 
that we are trying to avoid. 


Q85 Chairman: The only slight worry that this 
Committee would have is we know that the 
Ministers that we know in the Department for 
Education and Skills are more worried about those 
in other departments? 

Sir Alan Wilson: 1 can only say, Chairman, what we 
said at the very beginning in discussing this, that the 
departments are actually working well together. 


Q86 Jeff Ennis: Will metrics be consistent with the 
Government’s policy of enhancing the capacity of 
the UK’s research base in your opinion? 

Sir Alan Wilson: Again if I could start, Chairman. I 
think that the size of the research pot is, as it were, a 
massive argument in the next comprehensive 
spending review and I do not see any sense in which 
a change of methodology affects that decision, so I 
think decisions on support for research will be made 
in the usual way and, as I said in another context 
earlier, I think all the evidence is that there will be 
continuing support. 


Q87 Chairman: Do you agree with that, Professor 
Eastwood? 

Professor Eastwood: Yes, | do and I think one of the 
crucial outputs of the RAE and any successor to the 
RAE, is that within the framework of dual support 
institutions have resource that they themselves can 
direct, that is to say that they can invest strategically 
and I think it is getting that balance right which is 
critical and I think had we been doing this from 
scratch and been doing this in an environment where 
institutions were not yet used to research 
management, then I think we would all rightly be 
nervous, but I think what crucially this will result in 
as far as institutions are concerned, is QR as a block 
grant which will have a number of purposes. 
Underwriting the research base is one crucial 
purpose, but also enabling institutions to speculate, 
speculate in terms of blue-skies research is another 
and I think we need to preserve that and that is why 
I strongly welcome throughout all this the 
commitment on all sides to the maintenance of dual 
support which I think is absolutely critical. 


Q88 Jeff Ennis: Final question, Chairman. How 
would a metrics system achieve the high level of 
national and international buy-in that is required to 
maintain the reputation of UK _ research, the 
reputation that we currently enjoy? 

Sir Alan Wilson: J think in the end, as I said earlier 
in relation to our second place in the league table, it 
actually turns on bibliometrics and while in the past 
departments have promoted themselves in terms of 
their 5 or 5-star rating, that is not actually an 
international comparison because other countries, 
including the United States, are actually not doing 
that and there is no common base. If you actually 
look at bibliometric analyses of citations there is 
actually a common base, so I think as the expertise 
in that area develops it will actually improve our 
capability for international bench-marking. 


Q89 Stephen Williams: Can I just ask, Chairman, on 
a subject we have not really touched upon so far, you 
mentioned the fear that there may be five centres of 
excellence, if some people have their way perhaps a 
few more, perhaps two handfuls of centres of 
excellence, whereas we have over a hundred higher 
education institutions in this country. I helped 
launch a report commissioned by 35 universities 
largely from the CMU group, but some other 
universities outside that group as well, where they 
demonstrated they get research income from the 
private sector, the NHS, the EU, but very little 
indeed from the Funding Councils, partly because 
their research often cannot be cited because it is 
contract based, therefore it is not in the public 
domain. Is anything we are doing in this review 
going to look at how research funding can reach the 
post-92 universities or are we going to continue to 
have research funding concentrated in Russell 
Group, the 94 Group universities rather than spread 
out to other places that are still developing their 
research base? 

Sir Alan Wilson: 1 think it depends, Chairman, on 
the weighting of applied research in a funding 
formula and we have been very open in the 
consultation paper that this is a possibility and 
because we value that side of the research spectrum, 
very much value the contribution of the universities 
that Mr Williams is talking about and I think that 
weighting is a policy question for ministers in the 
future. If you actually look at the illustrative models 
on the web site there certainly are model runs which 
distribute more research funding to CMU 
universities because client research funding is being 
sO appropriately weighted in that illustrative 
funding formula, but at this stage it is not, as it were, 
for us to choose in policy terms, it is for our ministers 
down the line in institutional terms, it is for HEFCE 
down the line. 


Q90 Mr Chaytor: Sir Alan, one of the prominent 
arguments for a move to a metric system is the 
growth of interdisciplinary research. Why is there no 
reference to that in the consultation document? 

Sir Alan Wilson: | am sure there is, Chairman, when 
we re-stated the principles from the Next Steps paper 
and if it is missing, then I am sorry, but— 
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Mr Chaytor: I read it on the train coming down and 
I listened to what you said about it, and I skimmed 
through it again because I have got it here, and as far 
as I can see the word “interdisciplinary” does not 
appear once. 


Q91 Chairman: Sir Alan, I do not think you have got 
time to go right through it. Sir Alan, if you could 
drop us a line about that? 

Sir Alan Wilson: Yes, I certainly will.' 


Q92 Chairman: Can I say that this has been a good 
session. Is there anything you want to say before we 
close this session? 
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Sir Alan Wilson: No, I think I appreciate the 
questions from you and your colleagues, Chairman, 
it has been an interesting session, I agree with you on 
that. It is always valuable for us to be challenged and 
I appreciate that and we will learn things which we 
will take away and build into our thinking. 


Q93 Chairman: Professor Eastwood? 

Professor Eastwood: Can | thank you for the session 
and say I look forward to continuing the dialogue on 
this and other subjects. 

Chairman: We look forward to a long and happy 
relationship with you and you will have to live with 
us. Sir Alan, I have known you a long time in many 
roles and you have added lustre to all of them and 
can I wish you well in the new role and I think I said 
I will be at your new college only on Thursday night, 
so I shall suss out the catering for you. Good luck. 


Supplementary memorandum submitted by the Department for Education and Skills (DfES) 


At yesterday’s meeting of the Education and Skills Select Committee, it was claimed that the consultation 
document published on 13 June made no mention of the need to better recognise interdisciplinary research 
in the future assessment and funding arrangements. I promised to write to you in order to confirm whether 


or not this was the case. 


In fact, the consultation document acknowledges the importance of interdisciplinary research at 
paragraph 3.5 at again at paragraph 5.1. I should also point out that the terms of reference of the 
consultation working group, which are restated at Appendix I of the document, note specifically that any 
new system should “support interdisciplinary research”. 


July 2006 
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Memorandum submitted by the National Union of Students (NUS) 


THE RESEARCH ASSESSMENT EXERCISE CONSULTATION 


NUS believe the remit of the consultation is far too narrow. It focuses exclusively on how a metrics based 
RAE should work. This is a missed opportunity for a debate on some fundamental flaws in the RAE system. 
The consultation does not address the following important issues: 


The RAE only rewards university departments that already have a strong record in excellence, and 
therefore creates barriers to creating research excellence in the first place. This system creates a 
vicious cycle whereby the money always goes to the same departments, and other departments are 
not able to develop excellence in research. 


The RAE rewards excellence in research, and yet ignores another vital part of the work of 
universities, namely teaching students. This means that teaching is given less priority than 
research. 


QUESTIONS TO ASK 


Do you think that the remit of this consultation is too narrow, and is this a missed opportunity 
for a wider debate on the Research Assessment Exercise? What other problems are there which 
this consultation does not cover? 


Should excellence in teaching be given equal status and equal reward to excellence in research? 


How do we tackle the Catch 22 situation whereby it is the university departments with an already 
strong research capability who get the money to improve their research, whilst the departments 
who need more help and resources cannot access it? 


The consultation places great emphasis on universities accessing money from private companies. 
Should there be regulations on making ethical choices, to ensure that universities do not rely on 
wealthy oil companies or arms companies for their funding? Will this not compromise academic 
freedom and integrity? 


July 2006 
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